Portable device for optically measuring three-dimensional coordinates

ABSTRACT

A device for scanning and obtaining three-dimensional coordinates is provided. The device may be a hand-held scanner that includes a carrying structure having a front and reverse side, the carrying structure having a first arm, a second arm and a third arm arranged in a T-shape or a Y-shape. A housing is coupled to the reverse side, a handle is positioned opposite the carrying structure, the housing and carrying structure defining an interior space. At least one projector is configured to project at least one pattern on an object, the projector being positioned within the interior space and oriented to project the at least one pattern from the front side. At least two cameras are provided spaced apart from each other, the cameras being configured to record images of the object. The cameras and projector are spaced apart from each other by a pre-determined distance.

CROSS REFERENCE TO RELATED APPLICATIONS (IF APPLICABLE)

The present application claims the benefit of German Patent Application10 2014 113 015.4 filed on Sep. 10, 2014 entitled “Method for OpticallyScanning and Measuring an Environment” and is also acontinuation-in-part application of U.S. application Ser. No. 14/712,993filed on May 15, 2015 entitled “A Device and Method For OpticallyScanning and Measuring an Environment”, which is a nonprovisionalapplication of U.S. Provisional Application 62/161,461 filed on May 14,2015 entitled “A Device and Method For Optically Scanning and Measuringan Environment.” U.S. Ser. No. 14/712,993 further claims the benefit ofGerman Patent Application 10 2014 013 677.9 filed on Sep. 10, 2014. Thepresent application is further a continuation-in-part application ofU.S. application Ser. No. 14/722,219 filed on May 27, 2015 entitled“Device and Method for Optically Scanning and Measuring an Environmentand Method of Control”, which is a nonprovisional application of theaforementioned U.S. Provisional Application 62/161,461. U.S. Ser. No.14/722,219 claims the benefit of German Application 10 2014 013 678.7filed on Sep. 10, 2014. The contents of all of which are incorporated byreference herein in their entirety.

BACKGROUND OF THE INVENTION

The subject matter disclosed herein relates to a portable scanner, andin particular to a portable scanner having a display.

A portable scanner includes a projector that projects light patterns onthe surface of an object to be scanned. The position of the projector isdetermined by means of a projected, encoded pattern. Two (or more)cameras, the relative positions and alignment of which are known or aredetermined, can record images of the surface with a further, uncodedpattern. The three-dimensional coordinates (of the points of thepattern) can be determined by means of mathematical methods which areknown per se, such as epipolar geometry.

In video gaming applications, scanners are known as tracking devices, inwhich a projector projects an encoded light pattern onto the target tobe pursued, preferably the user who is playing, in order to then recordthis encoded light pattern with a camera and to determine thecoordinates of the user. The data are represented on an appropriatedisplay.

A system for scanning a scene, including distance measuring, mayinclude, in its most simplest form, a camera unit with two cameras, andillumination unit and a synchronizing unit. The cameras, which mayoptionally include filters, are used for the stereoscopic registrationof a target area. The illumination unit is used for generating a patternin the target area, such as by means of a diffractive optical element.The synchronizing unit synchronizes the illumination unit and the cameraunit. Camera unit and illumination unit can be set up in selectablerelative positions. Optionally, also two camera units or twoillumination units can be used.

BRIEF DESCRIPTION OF THE INVENTION

According to one aspect of the invention, a hand-held scanner forproducing 3D scans of an object in the environment if provided. Thescanner includes a carrying structure having a front side and a reverseside, the carrying structure having a first arm, a second arm and athird arm arranged in a T-shape or a Y-shape. A housing is coupled tothe reverse side having a handle, the handle being positioned oppositethe carrying structure, the housing and carrying structure defining aninterior space. At least one projector is configured to project at leastone pattern on the object, the at least one projector being positionedwithin the interior space and oriented to project the at least onepattern from the front side. At least two cameras are provided spacedapart from each other, the at least two cameras being configured torecord images of the object, the at least two cameras being disposedwithin the interior space and oriented to record images through thefront side, the at least two cameras and the at least one projectorbeing spaced apart by a pre-determined distance from each other by thecarrying structure.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWING

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 shows a perspective view of a hand-held scanner and of an objectin the environment;

FIG. 2 shows a view of the front side of the hand-held scanner;

FIG. 3 shows a view of the reverse side of the hand-held scanner;

FIG. 4 shows a top view of the hand-held scanner from above;

FIG. 5 shows a view of the hand-held scanner from the right side;

FIG. 6 shows a perspective view corresponding to FIG. 1 without housing;

FIG. 7 shows a representation of the control and evaluation device withdisplay;

FIG. 8 shows a representation corresponding to FIG. 7, with a smallerscale of the video image;

FIG. 9 shows a representation of the control and evaluation device withdisplay in accordance with another embodiment; and

FIG. 10 is a computer generated image of the display of FIG. 7 or FIG.9.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION OF THE INVENTION

In one embodiment, the carrying structure is stable mechanically andthermally, defines the relative distances and the relative alignments ofa camera and of a projector. The arrangement on a front side of the 3Dmeasuring device faces on the environment, has the advantage that thesedistances and alignments are not changed by a change of the shape of ahousing.

The term “projector” is defined to generally refer to a device forproducing a pattern. The generation of the pattern can take place bymeans of deflecting methods, such as generation by means of diffractiveoptical elements or micro-lenses (or single lasers), or by shadingmethods, for example the production by means of shutters, transparencies(as they would be used in a transparency projector) and other masks. Thedeflecting methods have the advantage of less light getting lost andconsequently a higher intensity being available.

Depending on the number of assemblies provided for distance measuring, acorresponding number of arms of the carrying structure is provided,which protrude from a common center located at the intersection of thearms. The assemblies, which may include a combination of cameras andprojectors, are provided in the area of the ends of the assigned arms.The assemblies may be arranged each on the reverse side of the carryingstructure. Their respective optics are directed through an assignedaperture in the carrying structure, so that the respective assembliesare operably oriented to face towards the environment from the frontside. A housing covers the reverse side and forms the handle part.

In one embodiment, the carrying structure consists of acarbon-reinforced or a glass-fiber-reinforced matrix of syntheticmaterial or ceramics (or of another material). The material provides forstability and a low weight and can, at the same time, be configured withviewing areas. A concave (spherical) curvature of the front side of thecarrying structure does not only have constructive advantages, but italso protects the optical components arranged on the front side when thefront surface of the 3D measuring device is placed on a work surface.

The projector produces the projected pattern, which may or may not bewithin the visible wavelength range. In one embodiment, the projectedpattern has a wavelength in the infrared range. The two cameras areconfigured to acquire images from light within this wavelength range,while also filtering out scattered light and other interferences in thevisible wavelength range. A color or 2D camera can be provided as thirdcamera for additional information, such as color for example. Suchcamera records images of the environment and of the object beingscanned. In an embodiment where the camera captures color, the pointcloud generated from the scanning process (herein referred to as the“3D-scan”) can have color values assigned from the color informationcontained in the color images.

During operation the 3D measuring device generates multiple 3D scans ofthe same scene, from different positions. The 3D scans are registered ina joint coordinate system. For joining two overlapping 3D scans, thereare advantages in being able to recognizable structures within the 3Dscans. Preferably, such recognizable structures are looked for anddisplayed continuously or, at least after the recording process. If, ina determined area, density is not at a desired level, further 3D scansof this area can be generated. A subdivision of the display used forrepresenting a video image and the (thereto adjacent parts of the)three-dimensional point cloud helps to recognize, in which areas scanshould still be generated.

In one embodiment, the 3D measuring device is designed as a portablescanner, i.e. it works at high speed and is of a size and weightsuitable for carrying and use by a single person. It is, however, alsopossible to mount the 3D measuring device on a tripod (or on anotherstand), on a manually movable trolley (or another cart), or on anautonomously moving robot, i.e. that it is not carried by theuser—optionally also by using another housing, for example without acarrying handle. It should be appreciated that while embodiments hereindescribe the 3D measuring device as being hand-held, this is forexemplary purposes and the claimed invention should not be so limited.In other embodiments, the 3D measuring device may also be configured asa compact unit, which are stationary or mobile and, if appropriate,built together with other devices.

Referring to FIGS. 1-6, a 3D measuring device 100 is provided asportable part of a device for optically scanning and measuring anenvironment of the 3D measuring device 100 with objects O. As usedherein, the side of the device 100 which faces the user shall bereferred to as the reverse side, and the side of the device 100 whichfaces the environment as the front side. This definition extends to thecomponents of the 3D measuring device 100. The 3D measuring device 100is provided (on its front side) visibly with a carrying structure 102having three arms 102 a, 102 b, 102 c. These arms give the carryingstructure 102 a T-shape or a Y-shape, i.e. a triangular arrangement. Thearea in which the three arms 102 a, 102 b, 102 c intersect and areconnected with each other, and from which the three arms 102 a, 102 b,102 c protrude, defines the center of the 3D measuring device 100. Fromthe user's view, the carrying structure 102 is provided with a left arm102 a, a right arm 102 b and a lower arm 102 c. In one embodiment, theangle between the left arm 102 a and the right arm 102 b is, forexample, approximately 150°+20°, between the left arm 102 a and thelower arm 102 c approximately 105°+10°. The lower arm 102 c is, in someembodiments, somewhat longer than the two other arms 102 a, 102 b.

The carrying structure 102 preferably is configured fromfiber-reinforced synthetic material, such as a carbon-fiber-reinforcedsynthetic material (CFC). In another embodiment, the carrying structure102 is made from carbon-fiber-reinforced ceramics or fromglass-fiber-reinforced synthetic material. The material renders thecarrying structure 102 mechanically and thermally stable and provides atthe same time for a low weight. The thickness of the carrying structure102 is considerably smaller (for example 5 to 15 mm) than the length ofthe arms 102 a, 102 b, 102 c (for example 15 to 25 cm). The carryingstructure 102 hence has a flat basic shape. In some embodiments, thearms 102 a, 102 b, 102 c, may include a reinforced back near the centerof the arm. It is, however, preferably not configured to be plane, butto be curved. Such curvature of the carrying structure 102 is adapted tothe curvature of a sphere having a radius of approximately 1 to 3 m. Thefront side (facing the object 0) of the carrying structure 102 isthereby configured to be concave, the reverse side to be convex. Thecurved shape of the carrying structure 102 is advantageous for providingstability. The front side of the carrying structure 102 (and in oneembodiment the visible areas of the reverse side) is configured to be aviewing area, i.e. it is not provided with hiders, covers, cladding orother kinds of packaging. The preferred configuration fromfiber-reinforced synthetic materials or ceramics is particularlysuitable for this purpose.

On the reverse side of the carrying structure 102, a housing 104 isarranged, which is connected with the carrying structure 102 within thearea of the ends of the three arms 102 a, 102 b, 102 c in a floatingway, by means of appropriate connecting means, for example by means ofrubber rings and screws with a bit of clearance. As used herein, afloating connection is one that reduces or eliminates the transmissionof vibration from the housing 104 to the carrying structure 102. In oneembodiment, the floating connection is formed by a rubber isolationmount disposed between the housing 104 and the carrying structure. Inone embodiment, an elastomeric seal, such as rubber, is disposed betweenthe outer perimeter of the carrying structure 102 and the housing 104.The carrying structure 102 and the housing 104 are then clamped togetherusing elastomeric bushings. The seal and bushings cooperate to form thefloating connection between the carrying structure 102 and the housing104. Within the area of the left arm 102 a and of the right arm 102 b,the edge of the housing 104 extends into the immediate vicinity of thecarrying structure 102, while the housing 104 extends from the center ofthe 3D measuring device 100 within the area of the lower arm 102 c, at adistance to the carrying structure 102, forming a handle part 104 g,bends off at the end of the handle part 104 g and approaches the end ofthe lower arm 102 c, where it is connected with it in a floating manner.The edge of the handle 104 g extends into the immediate vicinity of thecarrying structure 102. In some embodiments, sections of the carryingstructure 102 may include a reinforced back 102 r. The back 102 rprotrudes into the interior of the housing 104. The housing 104 acts asa hood to cover the reverse side of the carrying structure 102 anddefine an interior space.

The protective elements 105 may be attached to the housing 104 or to thecarrying structure 102. In one embodiment, the protective elements 105are arranged at the ends of and extend outward from the arms 102 a, 102b, 102 c to protect the 3D measuring device from impacts and from damageresulting thereof. When not in use, the 3D measuring device 100 can beput down with its front side to the bottom. Due to the concave curvatureof the front side, on the 3D measuring device will only contact the thesurface at the ends of the arms 102 a, 102 b, 102 c. In embodimentswhere the protective elements 105 are positioned at the ends of the arms102 a, 102 b, 102 c advantages are gained since the protective elements105 will provide additional clearance with the surface. Furthermore,when the protective elements 105 are made from a soft material forexample from rubber, this provides a desirable tactile feel for theuser's hand. This soft material can optionally be attached to thehousing 104, particularly to the handle part 104 g.

On the reverse side of the 3D measuring device 100, an control actuatoror control knob 106 is arranged on the housing 104, by means of which atleast optical scanning and measuring, i.e. the scanning process, can bestarted and stopped. The control knob 106 is arranged in the center ofthe housing 104 adjacent one end of the handle. The control knob 106 maybe multi-functional and provide different functions based on a sequenceof actions by the user. These actions may time based (e.g. multiplebutton pushed within a predetermined time), or space based (e.g. thebutton moved in a predetermined set of directions), or a combination ofboth. In one embodiment, the control knob 106 may be tilted in severaldirections in (e.g. left, right, up, down). In one embodiment, aroundthe control knob 106 there are at least one status lamp 107. In oneembodiment, there may be a plurality of status lamps 107. These statuslamps 107 may be used to show the actual status of the 3D measuringdevice 100 and thus facilitate the operation thereof. The status lamps107 can preferably show different colors (for example green or red) inorder to distinguish several status′. The status lamps 107 may be lightemitting diodes (LEDs).

On the carrying structure 102, spaced apart from each other at a defineddistance, a first camera 111 is arranged on the left arm 102 a (in thearea of its end), and a second camera 112 is arranged on the right arm102 b (in the area of its end). The two cameras 111 and 112 are arrangedon the reverse side of the carrying structure 102 and fixed thereto,wherein the carrying structure 102 is provided with apertures throughwhich the respective camera 111, 112 can acquire images through thefront side of the carrying structure 102. The two cameras 111, 112 arepreferably surrounded by the connecting means for the floatingconnection of the housing 104 with the carrying structure 102.

Each of the cameras 111, 112 have a field of view associated therewith.The alignments of the first camera 111 and of the second camera 112 toeach other are adjusted or adjustable in such a way that the fields ofview overlap to allow stereoscopic images of the objects O. If thealignments are fixed, there is a desired predetermined overlappingrange, depending on the application in which the 3D measuring device 100is used. Depending on environment situations, also a range of severaldecimeters or meters may be desired. In another embodiment, thealignments of the cameras 111, 112 can be adjusted by the user, forexample by pivoting the cameras 111, 112 in opposite directions. In oneembodiment, the alignment of the cameras 111, 112 is tracked andtherefore known to the 3D measuring device 100. In another embodiment,the alignment is initially at random (and unknown), and is thendetermined, such as be measuring the positions of the camera's forexample, and thus known to the 3D measuring device 100. In still anotherembodiment, the alignment is set and fixed during manufacturing orcalibration of the 3D measurement device 100.

The first camera 111 and the second camera 112 are preferablymonochrome, i.e. sensitive to a narrow wavelength range, for example bybeing provided with corresponding filters, which then filter out otherwavelength ranges, including scattered light. This narrow wavelengthrange may also be within the infrared range. In order to obtain colorinformation on the objects O, the 3D measuring device 100 preferablyincludes a 2D camera, such as color camera 113 which is preferablyaligned symmetrically to the first camera 111 and to the second camera112, and arranged in the center of the 3D measuring device 100, betweenthe cameras 111, 112. The 2D camera 113 may include an image sensor thatis sensitive to light in the visible wavelength range.

In order to illuminate the scene for the 2D camera, in the event ofunfavorable lighting conditions, at least one, in the illustratedembodiment a light source, such as four (powerful) light-emitting diodes(LED) 114 are provided. One radiating element 115 is associated witheach of the LEDs 114. The light emitted from the light-emitting diode114 is deflected in correspondence with the alignment of the 3Dmeasuring device 100, from the corresponding LED 114. Such a radiatingelement 115 can, for example, be a lens or an appropriately configuredend of a light guide. The (in the illustrated embodiment four) radiatingelements 115 are arranged equally around the color camera 113. Each LED114 is connected with the assigned radiating element 115 by means of onelight guide each. The LED 114 therefore can be structurally arranged ata control unit 118 of the 3D measuring device 100, such as by beingfixed on a board thereof.

In order to later have a reference for the images recorded by thecameras 111,112, 113, a sensor such as an inclinometer 119 is provided.In one embodiment, the inclinometer 119 is an acceleration sensor (withone or several sensitive axes), which is manufactured in a manner knownper se, as MEMS (micro-electro-mechanical system). As inclinometer 119,also other embodiments and combinations are possible. The data of the 3Dmeasuring device 100 each have (as one component) a gravitationdirection provided by the inclinometer 119.

During operation images are recorded by the first camera 111 and by thesecond camera 112. From these images three-dimensional data can bedetermined, i.e. 3D-scans of the objects O can be produced, for exampleby means of photogrammetry. The objects O, however, may have fewstructures or features and many smooth surfaces. As a result, thegeneration of 3D-scans from the scattered light of the objects O isdifficult.

To resolve this difficulty, a projector 121 may be used, which isarranged at the lower arm 102 c (in the area of its end). The projector121 is arranged within the interior space on the reverse side of thecarrying structure 102 and fixed thereto. The carrying structure 102 isprovided with an aperture through which the projector 121 can project apattern of light through the front side of the carrying structure 102.In one embodiment, the projector 121 is surrounded by the connectingmeans to provide a floating connection between the housing 104 with thecarrying structure 102. The projector 121, the first camera 111, and thesecond camera 112 are arranged in a triangular arrangement with respectto each other and aligned to the environment of the 3D measuring device100. The projector 121 is aligned in correspondence with the two cameras111, 112. The relative alignment between the cameras 111, 112 and theprojector 121 is preset or can be set by the user.

In one embodiment, the cameras 111, 112 and the projector 121 form anequilateral triangle and have a common tilt angle. When arranged in thismanner, and if the field of view of the cameras 111, 112 and theprojector 121 are similar, the centers of the field of view willintersect at a common point at a particular distance from the scanner100. This arrangement allows for a maximum amount of overlap to beobtained. In embodiments where the tilt or angle of the cameras 111, 112and projector 121 may be adjusted, the distance or range to theintersection of the fields of view may be changed.

If the user places 3D measuring device 100 on a surface on its frontside, i.e. with the front side to the surface, the concave curvature ofthe front side creates a gap between the cameras 111, 112, 113 and theprojector 121 from the surface, so that the respective lenses areprotected from damage.

The cameras 111, 112, 113, the projector 121, the control knob 106, thestatus lamps 107, the light-emitting diodes 114 and the inclinometer 119are connected with the common control unit 118, which is arranged insidethe housing 104. This control unit 118 can be part of a control andevaluation device which is integrated in the housing. In an embodiment,the control unit 118 is connected with a standardized communicationinterface at the housing 104, the interface being configured for awireless connection (for example Bluetooth, WLAN, DECT) as an emittingand receiving unit, or for a cable connection (for example USB, LAN), ifappropriate also as a defined interface, such as that described in DE 102009 010 465 B3, the contents of which are incorporated by referenceherein. The communication interface is connected with an externalcontrol and evaluation device 122 (as a further component of the devicefor optically scanning and measuring an environment of the 3D measuringdevice 100), by means of said wireless connection or connection bycable. In the present case, the communication interface is configuredfor a connection by cable, wherein a cable 125 is plugged into thehousing 104, for example at the lower end of the handle part 104 g, sothat the cable 125 extends in prolongation of the handle part 104 g.

The control and evaluation device 122 may include one or more processors122 a to carry out the methods for operating and controlling the 3Dmeasuring device 100 and evaluating the measured data. The control andevaluation device 122 may be a portable computer (notebook) or a tablet(or smartphone) such as that shown in FIGS. 7 and 8, or any external ordistal computer (e.g. in the web). The control and evaluation device 122may also be configured in software for controlling the 3D measuringdevice 100 and for evaluating the measured data. However, the controland evaluation device 122 may be embodied in separate hardware, or itcan be integrated into the 3D measuring device 100. The control andevaluation device 122 may also be a system of distributed components, atleast one component integrated into the 3D measuring device 100 and onecomponent externally. Accordingly, the processor(s) 122 a for performingsaid methods may be embedded in the 3D measuring device 100 and/or in anexternal computer.

The projector 121 projects a pattern X, which it produces, for exampleby means of a diffractive optical element, on the objects to be scanned.The pattern X does not need to be encoded (that is to saysingle-valued), but it is preferably uncoded, for example periodically,that is to say multivalued. The multi-valuedness is resolved by the useof the two cameras 111, 112, combined with the available, exactknowledge of the shape and direction of the pattern.

The uncoded pattern X is preferably a point pattern, comprising aregular arrangement of points in a grid. In the present invention, forexample, approximately one hundred times one hundred points areprojected at an angle of approximately 50° to a distance of approx. 0.5m to 5 m. The pattern X can also be a line pattern or a combined patternof points and lines, each of which is formed by tightly arranged lightpoints.

There is a relationship between the point density, the distance betweenthe projector 121 and the object O and the resolution that can beobtained with the produced pattern X. With diffractive patterngeneration, the light of one source is distributed over the pattern. Inthat case the brightness of the pattern elements depends on the numberof elements in the pattern when the total power of the light source islimited. Depending on the intensity of the light scattered from theobjects and the intensity of background light it may be determinedwhether it is desirable to have fewer but brighter pattern elements.Fewer pattern elements means the acquired point density decreases. Ittherefore seems helpful to be able to generate, in addition to patternX, at least one other pattern. Depending on the generation of thepatterns, a dynamic transition between the patterns and/or a spatialintermingling is possible, in order to use the desired pattern for thecurrent situation. In an embodiment, the projector 121 may produce thetwo patterns offset to each other with respect to time or in anotherwavelength range or with different intensity. The other pattern may be apattern which deviates from pattern X, such as an uncoded pattern. Inthe illustrated embodiment the pattern is a point pattern with a regulararrangement of points having another distance (grid length) to eachother.

For reasons of energy efficiency and eye protection, the projector 121produces the pattern X on the objects O only, when the cameras 111 and112 (and if available 113) record images of the objects O which areprovided with the pattern X. For this purpose, the two cameras 111, 112and the projector 121 are synchronized, i.e. coordinated internally witheach other, with regard to both, time and the pattern X used. Eachrecording process starts by the projector 121 producing the pattern X,similar to a flash in photography, and the cameras 111 and 112 (and, ifavailable 113) following with their records, more particularly theirpairs of records (frames), i.e. one image each from each of the twocameras 111, 112. The recording process can comprise one single frame(shot), or a sequence of a plurality of frames (video). Such a shot orsuch a video is triggered by means of the control knob 106. Afterprocessing of the data, each frame then constitutes a 3D-scan, i.e. apoint cloud in the three-dimensional space, in relative coordinates ofthe 3D measuring device 100.

The data furnished by the 3D measuring device 100 are processed in thecontrol and evaluation device 122, i.e. the 3D scans are generated fromthe frames. The 3D scans in turn are joined, i.e. registered in a jointcoordinate system. For registering, the known methods can be used, i.e.natural or artificial targets (i.e. recognizable structures) can belocalized and identified for example in overlapping areas of two 3Dscans, in order to determine the assignment of the two 3D scans by meansof corresponding pairs. A whole scene is thus gradually registered bythe 3D measuring device 100. The control and evaluation device 122 isprovided with a display 130 (display device), which is integrated orconnected externally.

One embodiment of the display 130 shown in FIG. 7 illustrates asubdivided image or subdivided screen. In this embodiment, the display130 is divided into a first display part 130 a and a second display part130 b. In the present embodiment, the first display part 130 a is a(rectangular) central part of the display 130, and the second displaypart 130 b is a peripheral area around the first display part 130 a. Inanother embodiment, the two display parts may be columns. In theembodiment illustrated in FIGS. 7-9, the first display part 130 a isshown as having a rectangular shape, however this is for exemplarypurposes and the claimed invention should not be so limited. In otherembodiments, the first display part 130 a may have other shapes,including but not limited to circular, square, trapezoid (FIG. 10),trapezium, parallelogram, oval, triangular, or a polygon having anynumber of sides. In one embodiment, the shape of the first display part130 a is user defined or selectable.

In the first display part 130 a a video live image VL is displayed, suchas that captured by 2D camera 113 for example. In the second displaypart 130 b, an image of the latest 3D scan (or a plurality of 3D scansthat have been registered) is displayed as at least part of a view ofthe three-dimensional point cloud 3DP. The size of the first displaypart 130 a may be variable, and the second display part 130 b isarranged in the area between the first display part 130 a and the border131 of the display 130. As video live image VL changes, such as when theuser moves the device 100, the image of the three-dimensional pointcloud 3DP changes correspondingly to reflect the change in position andorientation of the device 100.

It should be appreciated that the placement of the image of thethree-dimensional point cloud 3DP around the periphery of the video liveimage VL provides advantages in allowing the user to easily see whereadditional scanning may be required without taking their eyes off of thedisplay 130. In addition it may be desirable for the user to determineif the computational alignment of the current camera position to thealready acquired 3D data is within a desired specification. If thealignment is outside of specification, it would be noticed asdiscontinuities at the border between the image and thethree-dimensional point cloud 3DP. Referring now to FIG. 9, it can beseen that during a scanning operation some areas, such as areas 140, 142have a high density of points that allow for a representation of anobject at a desired accuracy level. The user will be able to observethat other areas, such as areas 144, 146 have lower point densities. Theuser may then determine whether additional scanning needs to beperformed. For example, area 144 may be a table top where a generallylow density of points may be acceptable. The user may determine thatother areas, such as area 146 for example, may require additionalscanning since the object has not been completely captured. FIG. 10illustrates a computer generated image of a scanning process, it shouldbe appreciated that the image of the three-dimensional point cloud 3DPis variable and continuously changes, which could make it difficult fora user to determine when additional scanning is needed. Thus the videolive image VL and the image of the three-dimensional point cloud 3DPcooperate to guide the user during the scanning process.

The image acquired by the camera 113 is a two-dimensional (2D) image ofthe scene. A 2D image that is rendered into a three-dimensional viewwill typically include a pincushion-shaped or barrel-shaped distortiondepending on the type of optical lens used in the camera. Generally,where the field of view (FOV) of the camera 113 is small (e.g. about 40degrees), the distortion is not readily apparent to the user. Similarly,the image of the three-dimensional point cloud data may appear distorteddepending on how the image is processed for the display. The point clouddata 3DP may be viewed as a planar view where the image is obtained inthe native coordinate system of the scanner (e.g. a spherical coordinatesystem) and mapped onto a plane. In a planar view, straight lines appearto be curved. Further, the image near the center-top and center-bottomedges (e.g. the poles) may be distorted relative to a line extendingalong the midpoint of the image (e.g. the equator). Further, there mayalso be distortions created by trying to represent a spherical surfaceon a rectangular grid (similar to the Mercator projection problem).

It should be appreciated that it is desired to have the images withinthe first display part 130 a appear to be similar to that in the seconddisplay part 130 b to provide a continuous and seamless image experiencefor the user. If the image of three-dimensional point cloud 3DP issignificantly distorted, it may make it difficult for the user todetermine which areas could use additional scanning. Since the planarimage of the point cloud data 3DP could be distorted relative to the 2Dcamera image, one or more processing steps may be performed on the imagegenerated from the point cloud data 3DP. In one embodiment, the field ofview (FOV) of the second display part 130 b is limited so that only thecentral portion of the planar image is shown. In other words, the imageis truncated or cropped to remove the highly distorted portions of theimage. Where the FOV is small (e.g. less 120 degrees), the distortion islimited and the planar view of the point cloud data 3DP will appear asdesired to the user. In one embodiment, the planar view is processed toscale and shift the planar image to provide to match the camera 113image in the first display part 130 a.

In another embodiment, the three-dimensional point cloud data 3DP isprocessed to generate a panoramic image. As used herein, the termpanoramic refers to a display in which angular movement is possibleabout a point in space (generally the location of the user). A panoramicview does not incur the distortions at the poles as is the case with aplanar view. The panoramic view may be a spherical panorama thatincludes 360 degrees in the azimuth direction and +/−45 degrees ion thezenith. In one embodiment the spherical panoramic view may be only aportion of a sphere.

In another embodiment, the point cloud data 3DP may be processed togenerate a 3D display. A 3D display refers to a display in whichprovision is made to enable not only rotation about a fixed point, butalso translational movement from point to point in space. This providesadvantages in allowing the user to move about the environment andprovide a continuous and seamless display between the first display part130 a and the second display part 130 b.

In one embodiment, the video live image VL in the first display part 130a and the image of the three-dimensional point cloud 3DP in the seconddisplay part 130 b match together seamlessly and continuously (withrespect to the displayed contents). A part of the three-dimensionalpoint cloud 3DP is first selected (by the control and evaluation device122) in such a way, as it is regarded from the perspective of the 2Dcamera 113 or at least from a position aligned with the 2D camera 113.Then, the selected part of the three-dimensional point cloud 3DP isselected in such a way that it adjoins continuously the video live imageVL. In other words, the displayed image of the three-dimensional pointcloud 3DP becomes a continuation of the video live image VL for theareas beyond the field of view of the 2D camera 113 on the left, on theright, top and bottom relative to the field of view of the 2D camera).As discussed above, the selected portion of the three-dimensional pointcloud 3DP may be processed to reduce or eliminate distortions. In otherembodiments, the representation may correspond to the representation ofa fish-eye lens, but preferably it is undistorted. The part of thethree-dimensional point cloud 3DP which is located in the area occupiedby the first display part 130 a, in other words the portion beneath orhidden by the video live image VL, is not displayed.

It should be appreciated that the density of the points in thethree-dimensional point cloud 3DP in the area where the first displaypart 130 a is located will not be visible to the user. Normally, thevideo live image VL is displayed using the natural coloring. However, inorder to indicate the density of the points in the area covered/behindby the video live image VL, the coloring of the video live image VL maybe changed artificially such as by overlaying for example. In thisembodiment, the artificial color (and, if appropriate, the intensity)used for representing the artificially colored video live image VLcorresponds to the density of the points. For example, a green coloringto the video live image VL may indicate a (sufficiently) high densitywhile a yellow coloring may be used to indicate a medium or low pointdensity (e.g. areas which still the scan data can be improved). Inanother embodiment, the distant-depending precision of the data pointscould be displayed using this color-coding.

To support the registration of the 3D scans, flags or marks 133 (FIG. 7and FIG. 8) may be inserted in the first display part 130 a to indicatestructures (i.e. possible targets) recognized by the control andevaluation device 122. The marks 133 may be a symbol, such as a small“x” or “+” for example. The recognizable structures can be points,corners, edges or textures of objects. The recognizable structures maybe found by the latest 3D scan or the video live image VL beingsubjected to the beginning of the registering process (i.e. to thelocalization of targets). The use of the latest video live image VLprovides advantages in that the registration process does not have to beperformed as frequently. If the marks 133 have a high density, it isconsidered to be a successful registration of the 3D scans. If, however,a lower density of the marks 133 is recognized, additional 3D scans maybe performed using a relatively slow movement of the 3D measuring device100. By slowing the movement of the device 100 during the scan,additional or higher density points may be acquired. Correspondingly,the density of the marks 133 may be used as a qualitative measure forthe success of the registration. Similarly, the density of the points ofthe three-dimensional point cloud 3DP may be used to indicate asuccessful scan. As discussed above, the density of points in the scanmay be represented by the artificial coloring of the video live imageVL.

The movement of the 3D measuring device 100 and processing of thecaptured frames may also be performed by a tracking function, i.e. the3D measuring device 100 tracks the relative movement of its environmentwith the methods used during tracking. If tracking gets lost, forexample, if the 3D measuring device 100 has been moved too fast, thereis a simple possibility of reassuming tracking. In this embodiment, thevideo live image VL as it is provided by the 2D camera 113 and the lastvideo still image from tracking provided by it may be representedadjacent to each other in a side by side arrangement on the display 130for the user. The user may then move the 3D measuring device 100 untilthe two video images coincide.

In one embodiment, the 3D measuring device 100 may be controlled basedon movements of the device 100. These movements or gestures by the usercan also be used for controlling the representation of the video imageVL or of the three-dimensional point cloud 3DP. In one embodiment, thescale of representation of the video image VL and/or of thethree-dimensional point cloud 3DP on the display 130 may depend on thespeed and/or acceleration of the movement of the 3D measuring device100. The term “scale” is defined as the ratio between the size (eitherlinear dimension or area) of the first display part 130 a and the sizeof the complete display 130, being denoted as a percentage.

A small field of view of the 2D camera 113 is assigned to a small scale.In the present embodiment with a subdivided display 130 with a centralfirst display part 130 a showing the video live image VL, this firstdisplay part 130 a then may be of smaller size than in the standardcase, and the second display part 130 b (about the periphery of thefirst display part 130 a) shows a bigger part of the three-dimensionalpoint cloud 3DP. A larger field of view is assigned to a large scale. Inone embodiment, the video live image VL may fill the whole display 130.

In the event of high speeds of movement of the 3D measuring device 100are detected, the scale of the representation may be configured smallerthan with low speeds and vice versa. Similarly, this may apply toaccelerations of the movement of the 3D measuring device 100. Forexample, the scale of the displayed image is reduced in the case ofpositive accelerations, and the scale is increased in the case ofnegative accelerations. The scale may also depend on a component of thespeed and/or acceleration of the movement of the 3D measuring device100, for example on a component which is arranged perpendicular orparallel to the alignment of the 3D measuring device 100. If the scaleis determined based on a component of the movement, parallel to thealignment (i.e. in the direction of the alignment), the scale can alsobe made dependent on the change of an average distance to objects O fromthe 3D measuring device 100.

In some embodiments, the change of the scale due to movement, astandstill of the movement of the 3D measuring device 100 or a thresholdspeed of movement value not being achieved can be used to record asequence of still images of the camera 113 with a low dynamic range.These images may be captured at low dynamic range but with differentexposure times or illumination intensities within the sequence togenerate a high dynamic range image therefrom.

In some embodiments, the direction of gravity may be defined at thebeginning of the registration process by a defined movement of the 3Dmeasuring device 100. This defined movement is carried out by the userby moving the device 100 in a vertical upward and downward movement forexample. In other embodiments, the direction of gravity may bedetermined from a set of statistics of all movements during theregistration process. A plane may be averaged from the coordinates ofthe positions taken by the device 100 while recording process along apath of movement through space. It is assumed that the averaged plane islocated horizontally in space, meaning that the direction of gravity isperpendicular to it. As a result, the use of inclinometer 119 fordetermining the direction of gravity may be avoided.

The evaluation of the coordinates of the positions may also be used fordetermining the kind of scene and, if appropriate, to offer differentrepresentations or operating possibilities. A path of movement around acenter location (with an alignment of the 3D measuring device 100oriented towards the interior), suggests an image of a single object O(object-centered image). Similarly, a path of movement that orients thedevice 100 towards the outside (and particularly longer straightsections of the path of movements) makes reference to an image of rooms.Thus, where it is determined that a room is being scanned, an image of afloor plan (top view) may be inserted into the display 130.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

1. A hand-held scanner for producing 3D scans of an object in anenvironment, the scanner comprising: a carrying structure having a frontside and a reverse side, the carrying structure having a first arm, asecond arm and a third arm arranged in a T-shape or a Y-shape; a housingcoupled to the reverse side having a handle, the handle being positionedopposite the carrying structure, the housing and the carrying structuredefining an interior space; at least one projector configured to projectat least one pattern on the object, the at least one projector beingpositioned within the interior space and oriented to project the atleast one pattern from the front side; at least two cameras are spacedapart from each other, the at least two cameras being configured torecord images of the object, the at least two cameras being disposedwithin the interior space and oriented to record images through thefront side, the at least two cameras and the at least one projectorbeing spaced apart by a pre-determined distance from each other by thecarrying structure.
 2. The hand-held scanner of claim 1 wherein: the atleast two cameras includes a first camera coupled to the first arm and asecond camera coupled to the second arm; and the at least one projectoris coupled to the third arm.
 3. The hand-held scanner of claim 2 furthercomprising: a first protective element coupled to the carrying structureat a first end of the first arm, the first protective element extendingoutward from the first end; a second protective element coupled to thecarrying structure at a second end of the second arm, the secondprotective element extending outward from the second end; and a thirdprotective element coupled to the carrying structure at a third end ofthe third arm, the third protective element extending outward from thethird end.
 4. The hand-held scanner of claim 3 further comprising acolor camera disposed at an intersection of the first arm, the secondarm and the third arm.
 5. The hand-held scanner of claim 4 furthercomprising a control actuator disposed on the housing opposite thecarrying structure.
 6. The hand-held scanner of claim 5 furthercomprising status lamps disposed about the control actuator.
 7. Thehand-held scanner of claim 1 wherein the carrying structure has aconcave curvature.
 8. The hand-held scanner of claim 7 wherein theconcave curvature of the carrying structure is a spherical curvaturehaving a radius between 1 to 3 meters.
 9. The hand-held scanner of claim1 wherein a first angle between the first arm and the second arm isbetween 130-170 degrees.
 10. The hand-held scanner of claim 9 whereinthe first angle is 150 degrees.
 11. The hand-held scanner of claim 9wherein a second angle between the first arm and the third arm is 95-115degrees.
 12. The hand-held scanner of claim 11 wherein the second angleis 110 degrees.
 13. The hand-held scanner of claim 11 wherein the thirdarm is longer than the first arm and the second arm.
 14. The hand-heldscanner of claim 1 wherein the housing is coupled to the carryingstructure by a floating connection.
 15. The hand-held scanner of claim14 wherein the floating connection comprises an elastomeric sealdisposed between an outer perimeter of the carrying structure and thehousing.
 16. The hand-held scanner of claim 1 wherein the carryingstructure is made from a fiber-reinforced synthetic material selectedfrom a group consisting of a carbon-fiber-reinforced synthetic material,a glass-fiber-reinforced synthetic material and fiber-reinforcedceramics.
 17. The hand-held scanner of claim 2 wherein the first camerahas a first field of view and the second camera has a second field ofview, the first field of view overlapping with the second field of view.18. The hand-held scanner of claim 17 wherein: at least one of the firstcamera and the second camera are pivotally coupled to the carryingstructure; and an amount of overlap between the first field of view andthe second field of view is adjustable by pivoting the first camera orthe second camera.
 19. The hand-held scanner of claim 1 furthercomprising a light source disposed within the interior space at anintersection of the first arm, the second arm and the third arm.
 20. Thehand-held scanner of claim 4 further comprising: a light source disposedwithin the interior space adjacent the color camera; and a radiatingelement coupled to the front side and disposed about the color camera,the radiating element configured to emit light from the light sourceonto the object.